feat: dont delete pages with jira tasks #207

samhotep · 2025-06-19T10:46:01Z

Problem Statement

The scheduled task to update webpages is breaking because there's an IntegrityError when deleting a webpage with an existing jira task.

The solution below avoids any deletion, and instead filters these pages from the requests for a tree. In the future, an endpoint can be added to specifically query pages which have been marked TO_DELETE (archived).

Done

Updated the scheduled tasks to not delete any webpages.
Updated the tree to return pages which aren't marked TO_DELETE

QA

Run the program, and open the main page.

docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres
dotrun

There should not be any database errors in the logs.

Fixes

2025-06-19T09:41:20.839Z [flask] The above exception was the direct cause of the following exception:
2025-06-19T09:41:20.839Z [flask] 
2025-06-19T09:41:20.839Z [flask] Traceback (most recent call last):
2025-06-19T09:41:20.839Z [flask]   File "/flask/app/webapp/scheduled_tasks.py", line 40, in load_site_trees
2025-06-19T09:41:20.839Z [flask]     site_repository.get_tree(no_cache=True)
2025-06-19T09:41:20.839Z [flask]   File "/flask/app/webapp/site_repository.py", line 278, in get_tree
2025-06-19T09:41:20.839Z [flask]     return self.get_new_tree()
2025-06-19T09:41:20.839Z [flask]   File "/flask/app/webapp/site_repository.py", line 212, in get_new_tree
2025-06-19T09:41:20.839Z [flask]     tree = self.create_webpages_for_tree(self.db, base_tree)
2025-06-19T09:41:20.839Z [flask]   File "/flask/app/webapp/site_repository.py", line 396, in create_webpages_for_tree
2025-06-19T09:41:20.839Z [flask]     self.__remove_webpages_to_delete__(db, tree)
2025-06-19T09:41:20.839Z [flask]   File "/flask/app/webapp/site_repository.py", line 362, in __remove_webpages_to_delete__
2025-06-19T09:41:20.839Z [flask]     db.session.execute(
2025-06-19T09:41:20.839Z [flask]   File "/lib/python3.10/site-packages/sqlalchemy/orm/scoping.py", line 779, in execute
. . .
2025-06-19T09:41:20.839Z [flask]     self.dialect.do_execute(
2025-06-19T09:41:20.839Z [flask]   File "/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
2025-06-19T09:41:20.839Z [flask]     cursor.execute(statement, parameters)
2025-06-19T09:41:20.839Z [flask] sqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) update or delete on table "webpages" violates foreign key constraint "jira_tasks_webpage_id_fkey" on table "jira_tasks"
2025-06-19T09:41:20.839Z [flask] DETAIL:  Key (id)=(214) is still referenced from table "jira_tasks".
2025-06-19T09:41:20.839Z [flask] 
2025-06-19T09:41:20.839Z [flask] [SQL: DELETE FROM webpages WHERE webpages.id = %(id_1)s]
2025-06-19T09:41:20.839Z [flask] [parameters: {'id_1': 214}]

webteam-app · 2025-06-19T10:46:05Z

Demo

Jenkins

demos.haus

muhammad-ali-pk · 2025-06-19T15:44:52Z

@samhotep After running the project and accessing localhost:8104/app in the browser, I get this error

$ ./entrypoint 0.0.0.0:${PORT}
$ watch -p 'static/client/**/*.{js,jsx,ts,tsx,scss}' -c 'yarn run build'
 * Tip: There are .env files present. Install python-dotenv to use them.
Watching started
INFO  [webapp] Connecting to Redis cache.
INFO  [googleapiclient.discovery_cache] file_cache is only supported with oauth2client<4.0.0
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
[2025-06-19 15:43:37 +0000] [173] [INFO] Starting gunicorn 23.0.0
[2025-06-19 15:43:37 +0000] [173] [INFO] Listening at: http://0.0.0.0:8104 (173)
[2025-06-19 15:43:37 +0000] [173] [INFO] Using worker: sync
[2025-06-19 15:43:37 +0000] [174] [INFO] Booting worker with pid: 174
[2025-06-19 15:43:37 +0000] [175] [INFO] Booting worker with pid: 175
INFO  [webapp] Connecting to Redis cache.
INFO  [webapp] Connecting to Redis cache.
INFO  [googleapiclient.discovery_cache] file_cache is only supported with oauth2client<4.0.0
INFO  [googleapiclient.discovery_cache] file_cache is only supported with oauth2client<4.0.0
Traceback (most recent call last):
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/flask/app.py", line 1514, in wsgi_app
    response = self.handle_exception(e)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/flask_cors/extension.py", line 194, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/flask_cors/extension.py", line 194, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/flask/app.py", line 915, in full_dispatch_request
    rv = self.preprocess_request()
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/flask/app.py", line 1291, in preprocess_request
    rv = self.ensure_sync(before_func)()
  File "/home/ubuntu/cs-canonical-com/webapp/models.py", line 216, in create_default_project
    get_or_create(db.session, Project, name="Default")
  File "/home/ubuntu/cs-canonical-com/webapp/models.py", line 49, in get_or_create
    instance = session.query(model).filter_by(**kwargs).first()
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/scoping.py", line 1680, in query
    return self._proxied.query(*entities, **kwargs)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2955, in query
    return self._query_cls(entities, self, **kwargs)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 276, in __init__
    self._set_entities(entities)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 288, in _set_entities
    self._raw_columns = [
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 289, in <listcomp>
    coercions.expect(
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/sql/coercions.py", line 388, in expect
    insp._post_inspect
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 1338, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 2724, in _post_inspect
    self._check_configure()
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 2401, in _check_configure
    _configure_registries({self.registry}, cascade=True)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 4214, in _configure_registries
    _do_configure_registries(registries, cascade)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 4255, in _do_configure_registries
    mapper._post_configure_properties()
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/mapper.py", line 2418, in _post_configure_properties
    prop.init()
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/interfaces.py", line 589, in init
    self.do_init()
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/relationships.py", line 1661, in do_init
    self._generate_backref()
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/relationships.py", line 2145, in _generate_backref
    self._add_reverse_property(self.back_populates)
  File "/home/ubuntu/cs-canonical-com/.venv/lib/python3.10/site-packages/sqlalchemy/orm/relationships.py", line 1594, in _add_reverse_property
    raise sa_exc.InvalidRequestError(
sqlalchemy.exc.InvalidRequestError: back_populates on relationship 'JiraTask.webpages' refers to attribute 'Webpage.jira_tasks' that is not a relationship.  The back_populates parameter should refer to the name of a relationship on the target class.

muhammad-ali-pk · 2025-06-19T16:00:10Z

@samhotep, webpages are not displaying their jira tasks.

muhammad-ali-pk · 2025-06-20T07:10:46Z

@samhotep

AttributeError: 'Webpage' object has no attribute 'jira_tasks'
INFO  [webapp.scheduled_tasks] Loading site tree for jp.ubuntu.com
INFO  [webapp] Connecting to Redis cache.
INFO  [webapp] Existing tree {'name': '', 'title': None, 'description': None, 'link': None, 'children': []}
ERROR  [webapp.scheduled_tasks] 'Webpage' object has no attribute 'jira_tasks'
Traceback (most recent call last):
  File "/home/ubuntu/cs-canonical-com/webapp/scheduled_tasks.py", line 40, in load_site_trees
    site_repository.get_tree(no_cache=True)
  File "/home/ubuntu/cs-canonical-com/webapp/site_repository.py", line 278, in get_tree
    return self.get_new_tree()
  File "/home/ubuntu/cs-canonical-com/webapp/site_repository.py", line 212, in get_new_tree
    tree = self.create_webpages_for_tree(self.db, base_tree)
  File "/home/ubuntu/cs-canonical-com/webapp/site_repository.py", line 378, in create_webpages_for_tree
    webpage_dict = self.__create_webpage_for_node__(
  File "/home/ubuntu/cs-canonical-com/webapp/site_repository.py", line 315, in __create_webpage_for_node__
    webpage_dict = convert_webpage_to_dict(webpage, owner, project)
  File "/home/ubuntu/cs-canonical-com/webapp/helper.py", line 96, in convert_webpage_to_dict
    webpage.jira_tasks
AttributeError: 'Webpage' object has no attribute 'jira_tasks'

samhotep · 2025-06-23T07:45:06Z

@muhammad-ali-pk I've updated the PR to not delete webpages on schedule. Instead, when a request is made, we filter all pages with the TO_DELETE status from the tree.

In the future, we should create a page (and endpoint) for viewing these 'archived' pages.

webapp/site_repository.py

github-actions · 2025-09-05T11:42:40Z

🎉 This PR is included in version 1.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

samhotep added Review: QA Needed Review: Code Needed labels Jun 19, 2025

samhotep requested a review from muhammad-ali-pk June 19, 2025 10:49

muhammad-ali-pk added Review: QA -1 and removed Review: QA Needed labels Jun 19, 2025

samhotep added Review: QA Needed and removed Review: QA -1 labels Jun 19, 2025

samhotep force-pushed the update-webpages-jira-field branch from 7c6ac42 to 839f1e5 Compare June 20, 2025 07:40

samhotep changed the title ~~feat: updated jira_tasks column to nullable~~ feat: dont delete pages with jira tasks Jun 20, 2025

muhammad-ali-pk approved these changes Jun 20, 2025

View reviewed changes

muhammad-ali-pk added Review: QA +1 Review: Code +1 and removed Review: QA Needed Review: Code Needed labels Jun 20, 2025

muhammad-ali-pk approved these changes Jun 23, 2025

View reviewed changes

webapp/site_repository.py Outdated Show resolved Hide resolved

samhotep added 4 commits June 24, 2025 10:40

feat: dont delete pages with jira tasks

8c9c2ef

feat: don't delete pages, sort from tree instead

4a98427

chore: lint

d88cc97

feat: removed redundant clause

c9b422c

samhotep force-pushed the update-webpages-jira-field branch from 2e37a3e to c9b422c Compare June 24, 2025 07:40

feat: separated clauses

0f897c9

samhotep merged commit 5f5508b into main Jun 24, 2025
7 of 8 checks passed

samhotep deleted the update-webpages-jira-field branch June 24, 2025 08:19

github-actions bot added the released label Sep 5, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: dont delete pages with jira tasks #207

feat: dont delete pages with jira tasks #207

Uh oh!

samhotep commented Jun 19, 2025 •

edited

Loading

Uh oh!

webteam-app commented Jun 19, 2025

Uh oh!

muhammad-ali-pk commented Jun 19, 2025

Uh oh!

muhammad-ali-pk commented Jun 19, 2025

Uh oh!

muhammad-ali-pk commented Jun 20, 2025

Uh oh!

samhotep commented Jun 23, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: dont delete pages with jira tasks #207

feat: dont delete pages with jira tasks #207

Uh oh!

Conversation

samhotep commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Statement

Done

QA

Fixes

Uh oh!

webteam-app commented Jun 19, 2025

Demo

Jenkins

demos.haus

Uh oh!

muhammad-ali-pk commented Jun 19, 2025

Uh oh!

muhammad-ali-pk commented Jun 19, 2025

Uh oh!

muhammad-ali-pk commented Jun 20, 2025

Uh oh!

samhotep commented Jun 23, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

samhotep commented Jun 19, 2025 •

edited

Loading